COMMENT ⊗   VALID 00001 PAGES
C REC  PAGE   DESCRIPTION
C00001 00001
C00002 ENDMK
C⊗;
∂17-Dec-87  1712	CL-Characters-mailer 	test    
Received: from IBM.COM by SAIL.STANFORD.EDU with TCP; 17 Dec 87  17:12:33 PST
Date: Thu, 17 Dec 87 11:35:57 PST
From: Thom Linden <baggins@ibm.com>
To: "X3J13: Character Subcommittee" <cl-characters@sail.stanford.edu>
Message-ID: <871217.113557.baggins@IBM.com>
Subject: test

  test of new router name

∂17-Dec-87  1809	CL-Characters-mailer 	mailbox name change, JEIDA interaction,  sub-topics  
Received: from IBM.COM by SAIL.STANFORD.EDU with TCP; 17 Dec 87  18:09:01 PST
Date: Thu, 17 Dec 87 17:46:49 PST
From: Thom Linden <baggins@ibm.com>
To: "X3J13: Character Subcommittee" <cl-characters@sail.stanford.edu>
Message-ID: <871217.174649.baggins@IBM.com>
Subject: mailbox name change, JEIDA interaction,  sub-topics
Subject: new mailbox router is now operational

As evidenced by the rejected message below,
cl-natural-languages is no more.  please use cl-characters.

Regards,
  Thom

------------------------------------------------------------


Date: 17 Dec 87 10:59:48
From: Mailer-Daemon at IBM.COM
To: BAGGINS

IBM.COM Mail Server unable to deliver the following mail to:cl-natural-languages
Reason:
Negative reply from Host:sail.stanford.edu
550 I don't know anybody named cl-natural-languages

           ** Text of Mail follows **
Date: Thu, 17 Dec 87 10:31:39 PST
From: Thom Linden <baggins@ibm.com>
To: "X3J13: Character Subcommittee"
    <cl-natural-languages@sail.stanford.edu>
Message-ID: <871217.103139.baggins@IBM.com>
Subject: mailbox name change, JEIDA interaction,  sub-topics

  Sometime soon, our router at stanford will change to cl-characters.
I'll broadcast as soon as I determine it is operational.

  My counterpart at the IBM Tokyo Research Lab, presented the IBM
character extensions proposal at a JEIDA meeting in Nov.  JEIDA knows
that this has not yet been discussed by our ANSI committee.

  Per our discussion at the Ft Collins meeting, I am inviting ISO&JEIDA
to join our conferencing (via the stanford router as soon as the
new name is in effect).

  Larry made the reasonable suggestion that we decide
on the sub-topics of the proposals and deal with each (initially)
somewhat independently.

  Hopefully, everyone has a copy of the proposal material by now!
Let me know if not and I will ship a copy asap.

  My stab at sub-topics is:

     Type hierarchy
        eg. thin-string

     Explicit character set manipulation
        eg. define-char-set

     Equivalence
        eg. define-equivalence-class

     I/O interface
        eg. print-width

     Character set (or subset) predicates
        eg. jcl:jis-char-p


  ?other suggestions?





Happy Holidays,
  Thom

∂21-Dec-87  1918	CL-Characters-mailer 	Network communications 
Received: from IBM.COM by SAIL.STANFORD.EDU with TCP; 21 Dec 87  10:59:00 PST
Date: Mon, 21 Dec 87 10:13:40 PST
From: Thom Linden <baggins@ibm.com>
To: "Dr. Takayasu Ito" <tito%aoba.aoba.tohoku.junet@relay.cs.net>,
    "Dr. Taiichi Yuasa" <yuasa%kurims.kurims.kyoto-u.junet@relay.cs.net>
cc: "X3J13: Character Subcommittee" <cl-characters@sail.stanford.edu>
Message-ID: <871221.101340.baggins@IBM.com>
Subject: Network communications

  The ANSI subcommittee handling character issues communicates
over the networks via a broadcast node (cl-characters) at Stanford.
You and/or the interested members of your committees are encouraged
to participate in these conversations.  If you inform me of the
appropriate net ids, I will have them added to the distribution
list.

Regards,
  Thom Linden

∂22-Dec-87  0600	CL-Characters-mailer 	Type hierarchy    
Received: from XEROX.COM by SAIL.STANFORD.EDU with TCP; 22 Dec 87  06:00:07 PST
Received: from Cabernet.ms by ArpaGateway.ms ; 22 DEC 87 06:01:12 PST
Date: 22 Dec 87 05:59 PST
From: Masinter.pa@Xerox.COM
Subject: Type hierarchy
In-reply-to: Thom Linden <baggins@ibm.com>'s message of Thu, 17 Dec 87 17:46:49
 PST
To: cl-characters@sail.stanford.edu
Message-ID: <871222-060112-6764@Xerox>

I've spent some time thinking about this:

I think it is a fundamental error, an unacceptable incompatible change, to
change the Common Lisp type STRING to be something other than (VECTOR
STRING-CHAR), as is suggested by all of the extant proposals.

I think one of our fundamental design goals is that the extended language
features being proposed be in fact extensions, in that current CL functions not
be in error.

Currently, you can assume after (TYPEP x 'STRING) that X can hold any
STRING-CHAR element. Allowing STRING to denote several different types of vector
whose element types are < STRING-CHAR would violate that assumption.

It isn't necessary to change STRING in an incompatible way, however. What is
really the intent of these proposals is to extend the various functions in CL
that currently take "STRING" to also allow them to take other types as well.

Suppose we define a new type

(defun character-vector-p (x) 
   (and (vectorp x) (subtypep (array-element-type x) 'string-char)))

(deftype character-vector () '(satisfies character-vector-p))..

Now extend all functions that take strings as input arguments and have them
accept any kind of character-vector. 

∂29-Dec-87  1449	CL-Characters-mailer 	Type hierarchy    
Received: from SCRC-RIVERSIDE.ARPA by SAIL.STANFORD.EDU with TCP; 29 Dec 87  14:49:26 PST
Received: from LM1.NSC.DIALNET.SYMBOLICS.COM by Riverside.SCRC.Symbolics.COM via DIAL with SMTP id 214218; 29 Dec 87 12:52:39 EST
Received: from LM2.NSC.Dialnet.Symbolics.COM by LM1.NSC.Dialnet.Symbolics.COM via CHAOS with CHAOS-MAIL id 19079; Tue 29-Dec-87 23:37:20 JST
Date: Tue, 29 Dec 87 23:37 JST
From: Carl Hoffman <CWH@LM1.NSC.Dialnet.Symbolics.COM>
Subject: Type hierarchy
To: Masinter.pa@Xerox.COM, CL-Characters@SAIL.Stanford.EDU
cc: Shiota@LM1.NSC.Dialnet.Symbolics.COM
In-Reply-To: <871222-060112-6764@Xerox>
Message-ID: <871229233714.3.CWH@LM2.NSC.Dialnet.Symbolics.COM>

    Date: 22 Dec 87 05:59 PST
    From: Masinter.pa@Xerox.COM

    I think it is a fundamental error, an unacceptable incompatible change, to
    change the Common Lisp type STRING to be something other than (VECTOR
    STRING-CHAR), as is suggested by all of the extant proposals.

Why do you feel that this is a fundamental error?  In the Symbolics Genera 7.1
implementation, the type STRING is the same as (OR (VECTOR STRING-CHAR) (VECTOR
CHARACTER)).  As far as I can tell, this hasn't caused a major compatibility
problem.  The CL programs I've seen which use strings have all run in the
Symbolics implementation without modification.

The Symbolics implementation returns the following results:

(TYPEP (MAKE-ARRAY 1 :ELEMENT-TYPE 'CHARACTER) '(VECTOR STRING-CHAR)) -> NIL
(TYPEP (MAKE-ARRAY 1 :ELEMENT-TYPE 'STRING-CHAR) '(VECTOR CHARACTER)) -> NIL
(TYPEP (MAKE-ARRAY 1 :ELEMENT-TYPE 'CHARACTER) 'STRING)               -> T
(STRINGP (MAKE-ARRAY 1 :ELEMENT-TYPE 'CHARACTER))                     -> T
(TYPEP (MAKE-ARRAY 1 :ELEMENT-TYPE 'STRING-CHAR) 'STRING)             -> T
(STRINGP (MAKE-ARRAY 1 :ELEMENT-TYPE 'STRING-CHAR))                   -> T

MAKE-ARRAY ELEMENT-TYPE 'STRING-CHAR returns an array which allocates 8 bits
per character.  MAKE-ARRAY ELEMENT-TYPE 'CHARACTER returns an array which
allocates 28 bits per character (16 bits of code, 8 bits of font, and 4 bits of
modifier).

I believe that the current plan is to change MAKE-ARRAY ELEMENT-TYPE
'STRING-CHAR to return an array which allocates 16 bits per character (for 16
bits of code) and to use MAKE-ARRAY ELEMENT-TYPE 'STANDARD-CHAR to do what is
currently done with MAKE-ARRAY ELEMENT-TYPE 'STRING-CHAR.

Incidentally, I haven't heard any discussion of Moon's proposal that we simply
use the type STANDARD-CHAR to mean "lowest overhead character storage class"
rather than introducing a new type THIN-CHAR or INTERNAL-THIN-CHAR.

    Currently, you can assume after (TYPEP x 'STRING) that X can hold any
    STRING-CHAR element. Allowing STRING to denote several different types of vector
    whose element types are < STRING-CHAR would violate that assumption.

Why not just declare that assumption obsolete, and replace it with the
assumption that if (TYPEP X '(VECTOR STRING-CHAR)) then X can hold any
STRING-CHAR element.  Can you give me some examples of code which make use of
your assumption?

    It isn't necessary to change STRING in an incompatible way, however. What is
    really the intent of these proposals is to extend the various functions in CL
    that currently take "STRING" to also allow them to take other types as well.

That is only part of the intent.  It is also important that the following
forms return T.  (Assume that # represents a Japanese character.)

  (STRINGP "#")
  (TYPEP "#" 'STRING)
  (TYPEP (CHAR "#" 0) 'STRING-CHAR)

If the above forms do not return T, then many CL programs originally written to
handle only standard characters will not work when running in an environment
which has Japanese characters.  A major goal of this proposal is to allow these
programs to run without modification.  I can show you many programs which
require that the above forms return T.

    Suppose we define a new type

    (defun character-vector-p (x) 
       (and (vectorp x) (subtypep (array-element-type x) 'string-char)))

    (deftype character-vector () '(satisfies character-vector-p))..

    Now extend all functions that take strings as input arguments and have them
    accept any kind of character-vector. 

If you replace STRING-CHAR in your example with CHARACTER, then this is exactly
the same as what Symbolics has already done with the STRINGP function and the
STRING data type.


∂06-Jan-88  2217	CL-Characters-mailer 	Re: Type hierarchy
Received: from XEROX.COM by SAIL.STANFORD.EDU with TCP; 6 Jan 88  22:17:01 PST
Received: from Cabernet.ms by ArpaGateway.ms ; 06 JAN 88 22:17:43 PST
Date: 6 Jan 88 22:16 PST
From: Masinter.pa@Xerox.COM
Subject: Re: Type hierarchy
In-reply-to: Carl Hoffman <CWH@LM1.NSC.Dialnet.Symbolics.COM>'s message of Tue,
 29 Dec 87 23:37 JST
To: CWH@LM1.NSC.Dialnet.Symbolics.COM
cc: Masinter.pa@Xerox.COM, CL-Characters@SAIL.Stanford.EDU,
 Shiota@LM1.NSC.Dialnet.Symbolics.COM
Message-ID: <880106-221743-6432@Xerox>

I've composed several replies and not sent them. My time is getting tight so I
have to send something. The problem is, can you have something that is a string
for which it is illegal to store a string-char into it?  No, in SCL. But if you
allow (vector standard-char) to also be a subtype of string, then you can have
vectors that can only hold standard-char and not string-char.

However, on even further reflection, there are many "read-only" strings, e.g.,
strings as program constants, for which it is an error to store *anything*. 

If we remove char-bits and char-font, we can get rid of the distinction between
string-char and character. This would be an improvement.

Most of the stuff in CLtL about the string type can in fact simply be removed,
while simplifying the language.

∂07-Jan-88  2036	CL-Characters-mailer 	X3J13 meeting in March 
Received: from IBM.COM by SAIL.STANFORD.EDU with TCP; 7 Jan 88  20:36:20 PST
Date: Wed, 06 Jan 88 09:58:06 PST
From: Thom Linden <baggins@ibm.com>
To: "X3J13: Character Subcommittee" <cl-characters@sail.stanford.edu>
Message-ID: <880106.095806.baggins@IBM.com>
Subject: X3J13 meeting in March

  I have arranged for our subcommittee to meet at the IBM Almaden
Research Centre on 14,15,18 March.  Please let me know if this
poses any difficulties.  Also, please let me know if your travel
arrangements or other commitments prevent your attending all or
part.

  ARC is south of Palo Alto, roughly a 40 to 50 min commute.
I would suggest our meetings begin at 10am to allow missing most
of the morning freeway congestion.  I'll provide more detailed
directions later.

Regards,
  Thom

∂07-Jan-88  2036	CL-Characters-mailer 	subcommittee mailing list   
Received: from IBM.COM by SAIL.STANFORD.EDU with TCP; 7 Jan 88  20:36:05 PST
Date: Tue, 05 Jan 88 21:42:40 PST
From: Thom Linden <baggins@ibm.com>
To: "Richard P. Gabriel" <rpg@sail.stanford.edu>
cc: "Dr. Takayasu Ito" <ito%ito.aoba.tohoku.junet@relay.cs.net>,
    "X3J13: Character Subcommittee" <cl-characters@sail.stanford.edu>
Message-ID: <880105.214240.baggins@IBM.com>
Subject: subcommittee mailing list

Dick,
  Please add the following individuals to the character subcommittee
mailing list:

  Yuasa:     yuasa@tutics.tut.junet
  Umemura:   umemura@nuesun.NTT.junet
  Kurokawa:  KUROKAWA%jpntscvm.bitnet%wiscvm.wisc.edu
  Yasumura:  yasumura@harl86.harl.hitachi.junet

Regards,
  Thom

∂07-Jan-88  2036	CL-Characters-mailer 	Comments on IBM Proposal from Dave Unitas (LUCID)    
Received: from IBM.COM by SAIL.STANFORD.EDU with TCP; 7 Jan 88  20:36:41 PST
Date: Wed, 06 Jan 88 12:50:38 PST
From: Thom Linden <baggins@ibm.com>
To: "X3J13: Character Subcommittee" <cl-characters@sail.stanford.edu>
Message-ID: <880106.125038.baggins@IBM.com>
Subject: Comments on IBM Proposal from Dave Unitas (LUCID)

  I have attached some comments on the proposal compiled by
Dave Unitas at LUCID.

  Both A and C seem to be good suggestions.


--------------------------------------------------------------------------------

A. Each character set is identified by its Character Set Name, a symbol,
   and an associated Character Set Number, a positive integer. (Replace
   CSID by Character Set Name or Character Set Number throughout the
   document).

   Replace char-split and char-join with:

     char-code-point char-code

   takes a character code and returns the component code-point.

     char-code-set char-code

   takes a character code and returns the component character set.

     make-char-code code-point &optional (character-set 0)

   takes a code-point and an optional character set and returns the
   character code.  The character set may be specified either as a
   Character Set Name of Character Set Number.


   Rename define-char-set to be define-character-set.  Make the arguments
   keywords rather than positionals.  If character-set-number is not
   specified, it is assigned from an available character set number
   below character-set-limit.

   Note:  Lucid as a whole is as yet undecided about whether user-
   defined character sets are generally useful enough to need to be
   included in the language.

B. We are still unsure about whether the type system should be extended
   to include extended strings of a particular character set or sets.

C. When printing an exted character set to a stream which only accepts
   base characters, it is printed in the form

   #\name:xxxx

   where name identifies the character set of the character, and xxxx
   is the code-point of the character in hex.  Strings containing
   extended characters are printed in the following form when written
   to a base-character only stream:

   #( char0 char1 char2 ...)

   with charn as above, following the standard Common Lisp vector
   printing convention.

∂10-Jan-88  0010	CL-Characters-mailer 	Re: Comments on IBM Proposal from Dave Unitas (LUCID)
Received: from Xerox.COM by SAIL.Stanford.EDU with TCP; 10 Jan 88  00:10:01 PST
Received: from Cabernet.ms by ArpaGateway.ms ; 10 JAN 88 00:10:39 PST
Date: 10 Jan 88 00:09 PST
From: Masinter.pa@Xerox.COM
Subject: Re: Comments on IBM Proposal from Dave Unitas (LUCID)
In-reply-to: Thom Linden <baggins@ibm.com>'s message of Wed, 06 Jan 88 12:50:38
 PST
To: baggins@ibm.com
cc: cl-characters@sail.stanford.edu
Message-ID: <880110-001039-3219@Xerox>

I think one of the problems with the discussion so far is that we've not agreed
really on the fundamental issue of whether the standard is for an optional
extension or for a required part of the standard.

For the record, I think that we should be designing things that are a required
part of every Common Lisp implementation. That is, every function, variable,
etc. in our standard should be in every Common Lisp implementation.  In some
implementations, the characters they work on are of course only 7 or 8-bit
ASCII, but all of the functions are there, and if the implementation has more
characters or Japanese characters, the same code will work.

If this is a required part of Common Lisp, we should try to keep to a minimum
the number of new functions, variables, and behaviors we expect from a Common
Lisp implementation. 

I don't think that the introduction of new functions and variables for dealing
with character sets really fits that criteria. The only situation where allowing
exposure to multiple character sets within a single implementation makes sense
is one in which the host operating system does not contain facilities to do
character set translation, and yet the programmer is unwilling (using binary
read-byte write-byte) to do that character set translation directly. This seems
like an extremely narrow application domain for the dozens of functions and
variables which exist in the IBM proposal.

= = = = 

As a side note, the IBM proposal contains a fairly serious design flaw: the
Common Lisp design is generally careful to avoid having dynamically modifiable
global state that isn't rebindable; e.g., although you can change macro
characters, all changes happen to *readtable*, etc.  Yet the character code
equivalency tables in the IBM proposal are global and not yet bindable. Even if
this isn't part of the standard but an internal library for you, you should fix
it.

= = = = = =
About the type system: the discussion on Common-LIsp@sail.stanford.edu on array
element type upgrading is relevant to the type hierarchy here. Suppose arrays
remember their element type. Redefine (stringp x) = (and (vectorp x) (subtypep
(array-element-type x) 'character)). 

If you want to make a string that consists of only (capital) vowels,, you can
say
(make-array 10 :element-type '(member #\A #\E #\I #\O #\U)).



= = = = = = =
Re: "C. When printing an exted character set to a stream which only accepts
   base characters, it is printed in the form ... Strings containing
   extended characters are printed in the following form when written
   to a base-character only stream ..."

how are symbols that contain extended characters printed?

What happens when you call PRINC (which is supposed to not include the #\)?

I think this is a bad design. If you want to write extended characters on a base
stream, you should design a character-by-character encoding with escape
characters, and have the write-char primitive for the base stream turn the
extended characters (and the excape) into an escaped character sequence. These
alternative print sequences only handle a small percentage of the situations.